feat(strategy): rrqr-nystrom -- redundancy-avoiding landmark selection for decaying-spectrum data#226
Draft
RenzoMXD wants to merge 1 commit into
Draft
feat(strategy): rrqr-nystrom -- redundancy-avoiding landmark selection for decaying-spectrum data#226RenzoMXD wants to merge 1 commit into
RenzoMXD wants to merge 1 commit into
Conversation
RenzoMXD
marked this pull request as ready for review
July 14, 2026 21:37
Contributor
|
Gate chain passed. This PR is queued for the next batched GPU evaluation window. |
RenzoMXD
force-pushed
the
feat/nystrom-three-way
branch
from
July 15, 2026 02:56
56d7c4f to
a7dd9dd
Compare
RenzoMXD
marked this pull request as draft
July 15, 2026 02:57
Contributor
|
Gate chain passed. This PR is queued for the next batched GPU evaluation window. |
RenzoMXD
force-pushed
the
feat/nystrom-three-way
branch
from
July 15, 2026 04:09
a7dd9dd to
9615bcf
Compare
…n for decaying-spectrum data Replaces this PR's block-krylov content. Repeat GPU testing (5 fresh-seed runs, --data-rank 256 --rank-m 256) showed block-krylov was a regression versus the retired nystrom-iter attempt on both axes at once: worse accuracy (0.843-0.864 vs nystrom-iter's 0.875-0.891) and worse latency (~2.5x slower than exact vs ~1.7x). Splitting a fixed per-part width budget across Krylov depth traded away direct landmark breadth for GEMM-refined columns and lost more than it gained -- the second consecutive confirmation that spending the fixed M budget on extra GEMMs against the full n x n operand burns latency faster than it buys accuracy at this scale. Before implementing a third attempt, checked the actual premise numerically first this time. A "leverage-score nystrom" idea (weight landmark draws by column importance) was ruled out by computing true leverage scores of this project's decaying-spectrum test fill directly: they come out essentially uniform (std/mean ~0.08, matching a random subspace), because the fill's k^-alpha decay is applied to an abstract rank-direction axis that a dense Gaussian mix then spreads evenly across every physical column -- there is no per-column importance signal to weight by. What a second numeric check found instead: uniform-random landmark draws are wasteful in a different way -- they can by chance include redundant (near-parallel) columns. Oversampling a small candidate pool and greedily selecting the least-redundant subset via column-pivoted Gram-Schmidt (RRQR-style) beat plain uniform sampling's column-space recovery in 8/8 trials (mean rel-err 0.120 vs 0.132). This only needs a small oversampled candidate pool, not a GEMM against the full operand, so cost is O(n w^2) rather than O(n^2 w) -- roughly two orders of magnitude cheaper at n=8192 than every GEMM-refinement approach tried on this PR so far, directly avoiding the failure mode that sank both of them.
RenzoMXD
force-pushed
the
feat/nystrom-three-way
branch
from
July 15, 2026 05:56
9615bcf to
42d9d33
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR kind
DRAFT -- GPU scorecard pending. Replaces this PR's third attempt (
block-krylov). No GPU available right now for this content; opening as a draft to avoid the missing-scorecard auto-close.Why the previous content is gone
block-krylovwas repeat-tested (5 fresh-seed GPU runs, pinned decaying-spectrum regime--data-rank 256 --rank-m 256) and turned out to be a regression versus the retirednystrom-iterattempt on both axes at once:block-krylovnystrom-iter(retired)nystromSplitting a fixed per-part width budget across Krylov depth traded away direct landmark breadth for GEMM-refined columns and lost more than it gained -- the second consecutive confirmation that spending the fixed
Mbudget on extra GEMMs against the fulln x noperand burns latency faster than it buys accuracy at this scale, no matter how the GEMMs are arranged.Checked the premise numerically before writing a third implementation
Before touching GPU time again, two cheap NumPy checks (no GPU needed):
decaying-spectrumfill directly: they come out essentially uniform (std/mean ~0.08, matching a random subspace). The fill'sk^-alphadecay is applied to an abstract rank-direction axis that a dense Gaussian mix then spreads evenly across every physical column -- there is no per-column importance signal to weight by on this synthetic data. Any cheap leverage-score proxy (e.g. raw column norm) would just be chasing sampling noise, not real structure.oversample * wlandmarks, still just a memory gather) and greedily picking the least-redundant subset via column-pivoted Gram-Schmidt (RRQR-style) beat plain uniform sampling's column-space recovery in 8/8 trials (mean rel-err 0.120 vs 0.132, tight std bands, no overlap).Summary (new content)
New transform
rrqr-nystrom:{col(A), row(A), row(B)}as every prior attempt on this PR (#156's redundancy proof still holds)oversample * w(defaultoversample=4) uniform landmark columns/rows per part -- still a pure memory gather, same mechanism asnystromwof them via column-pivoted Gram-Schmidt: repeatedly pick the remaining candidate with the largest residual norm after projecting out everything already chosenCrucially, the pivoted selection runs on the small oversampled pool, not a GEMM against the full
n x noperand -- cost isO(n w²), notO(n² w), roughly two orders of magnitude cheaper at n=8192 thannystrom-iter/block-krylov's GEMM-refinement approach. This directly targets the failure mode that sank both previous attempts on this PR, rather than trying a third variant of the same GEMM-refinement idea.Verified so far (CPU, numpy only -- not a claim of GPU dominance)
test_rrqr_nystrom_beats_plain_nystrom_on_decaying_spectrumreproduces the pre-implementation numeric check as an automated test: pivoted selection from an oversampled pool spanscol(A)measurably better than an equal-size uniform draw ondecaying-spectrumdata. 9 tests total: registration, input validation (including graceful handling ofoversample=1), orthonormality, seed determinism, exact low-rank recovery, and honest (non-zero)basis_flops.CPU-safe validation
GPU Result (required for feat PRs only)
Pending -- will attach before marking ready for review. Plan: same pinned decaying-spectrum regime used for both prior honest negative results on this PR (
--fill decaying-spectrum --data-rank 256 --rank-m 256), comparingrrqr-nystromagainstnystrom(isolates the selection mechanism's contribution) andrsvd, 5+ fresh seeds -- not a single run.Scope
strategy/transforms.py+strategy/README.md+strategy/tests/test_rrqr_nystrom.pyonly (Open -- "the main event" zone per CONTRIBUTING.md); new transform, no API change, no protected-path edits.Checklist
Co-authored-byfooters for coding agents